Splunk Forwarder Status: Monitoring and Troubleshooting

In Splunk version 6.2, the Deployment Monitor App provides a comprehensive view of all forwarders in your system. This article will guide you through the process of monitoring and troubleshooting your Splunk forwarders.

Monitoring Forwarder Status

To monitor the status of your forwarders, use the following search:

```

| table hostname owner mostRecentConnection |

fillnull '- missing -' mostRecentConnection |

eval mostRecentConnection=strftime(mostRecentConnection,'%x %X')|

stats latest(_time) as mostRecentConnection by hostname|

join hostname 'search index=_internal source=*metrics.log group=tcpin_connections'

```

This search provides information on all forwarders in your system, indicating whether they have been heard from during the reporting period. The `mostRecentConnection` field displays the last time each forwarder was seen.

Identifying Missing Forwarders

To identify missing forwarders, use the following search:

```

| inputlookup host_lookup

```

This search uses a lookup table to identify hosts that are not responding. You can set up this lookup by adding a field lookup in your Splunk dashboard.

Creating a Lookup Table

A lookup table is essential for identifying missing forwarders. Create a table containing the expected hosts, including fields such as hostname, IP address, and owner. This allows you to track changes in your forwarder landscape over time.

Troubleshooting Forwarder Issues

If a host used to be reporting but stopped yesterday, use the following search:

```

| fieldformat Hour=strftime(Hour,'%x %X')|

fieldformat total_KB=tostring(total_KB,'commas')|

rename os as OS|

eval stats sum(kb) as total_KB by Hour connectType Ver sourceIp sourceHost MachType os Indexer destPort|

fields connectType sourceIp sourceHost destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server Ver MachType os|

eval version=if(isnull(version),'pre 4.2',version)|

eval connectType=case(fwdType='uf','univ fwder', fwdType='lwf', 'lightwt fwder',fwdType='full', 'heavy fwder', connectType='cooked' or connectType='cookedSSL','Splunk fwder', connectType='raw' or connectType='rawSSL','legacy fwder')|

eval sourceHost=if(isnull(hostname), sourceHost,hostname)|

index=_internal source=*metrics.log group=tcpin_connections

```

This search provides detailed information on forwarder connections, including the number of bytes sent and received. Use this data to identify potential issues with your forwarders.

Embedded Search

To make it easy for users to access the latest search results, set up an embedded search. This creates a link that allows users to view the search results even if they do not have access to the underlying data.

Scheduled Search

Save the search as a scheduled search to run periodically (e.g., once per hour). This ensures that your forwarder status is monitored and updated regularly.

Reporting Forwarder Details

To display the forwarder details in a report, use the following search:

```

| metadata type=hosts index=*|

eval latestEventSeen=strftime(lastTime,'%x %X')|

fields host latestEventSeen

```

This search provides information on each host's latest event seen. Use this data to create reports that display the forwarder details.

In conclusion, monitoring and troubleshooting your Splunk forwarders is crucial for ensuring the smooth operation of your system. By using the techniques outlined in this article, you can identify issues with your forwarders and take corrective action.

Splunk forwarder status