Just for reference… here are some searches extracted from the Deployment Monitor application (only slightly modified) and other useful ones. I know there are apps for this, obviously DM that I extracted some searches from. But who knows, sometimes these apps may not work.
The great ppl @ Splunk have also documented these searches and more:
http://www.splunk.com/wiki/Community:TroubleshootingIndexedDataVolume
http://www.splunk.com/wiki/Deploy:Splunk_Metric_Reports
Usage by source:
index=”_internal” source=”*license_usage.log” | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by source | eval mbytes=((bytes/1024)/1024) | fields source lastReceived mbytes | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by source | stats max(lastReceived) as lastReceived first(mbytes) as “MB Used” by source
Usage by host:
index=”_internal” source=”*license_usage.log” | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | stats sum(bytes) as bytes max(lastReceived) as lastReceived by host | eval mbytes=((bytes/1024)/1024) | fields host mbytes lastReceived | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by host | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by host
Usage by sourcetype:
index=”_internal” source=”*license_usage.log” | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server | eval mbytes=((bytes/1024)/1024) | stats max(lastReceived) as lastReceived sum(mbytes) as mbytes by mysourcetype | stats max(lastReceived) as lastReceived first(mbytes) as MBytes by mysourcetype | eval lastConnected=lastReceived | eval status = if(isnull(bytes) or lastConnected<(info_max_time-900),”missing”,”active”)
Forwarder statistics:
index=”_internal” source=”*metrics.log” group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”Universal Forwarder”, fwdType==”lwf”, “Light Weight Forwarder”,fwdType==”full”, “Splunk Indexed”, connectionType==”cooked” or connectionType==”cookedSSL”,”Splunk Forwarder”, connectionType==”raw” or connectionType==”rawSSL”,”Legacy Forwarder”) | eval build=if(isnull(build),”n/a”,build) | eval version=if(isnull(version),”pre 4.2″,version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),”n/a”,os)| eval arch=if(isnull(arch),”n/a”,arch) | eval my_splunk_server = splunk_server | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps my_splunk_server build version os arch | eval lastReceived = if(kb>0, _time, null) | stats first(sourceIp) as sourceIp first(connectionType) as connectionType first(sourcePort) as sourcePort first(build) as build first(version) as version first(os) as os first(arch) as arch max(_time) as lastConnected max(lastReceived) as lastReceived sum(kb) as kb avg(tcp_eps) as avg_eps by sourceHost | stats first(sourceIp) as sourceIp first(connectionType) as connectionType first(sourcePort) as sourcePort first(build) as build first(version) as version first(os) as os first(arch) as arch max(lastConnected) as lastConnected max(lastReceived) as lastReceived first(kb) as KB first(avg_eps) as eps by sourceHost | eval status = if(isnull(KB) or lastConnected<(info_max_time-900),”missing”,if(lastConnected>(lastReceived+300) or KB==0,”quiet”,”active”)) | sort sourceHost
Amount indexed by day:
index=_internal source=*metrics.log group=per_index_thruput series!=_* | eval totalGB = (kb/1024)/1024 | timechart span=1d sum(totalGB)
Per Index statistics (event count, KB indexed, kbps, eps):
index=_internal sourcetype=splunkd source=*metrics* “group=per_index_thruput” NOT series=”_thefishbucket” | eval events=eps*kb/kbps | replace “default” with “main” in series | stats sum(events) as events, sum(kb) as kb, median(kbps) as kbps, median(eps) as eps by series | eval events=round(events,0) | eval kb=round(kb,1)