Splunk HEC

Why: Splunk Cloud uses a different host for HEC than the web UI. Sending events to the wrong host fails quietly with DNS or TLS errors that look like bugs.

The HEC URL follows the same pattern as your web UI URL but the port and (sometimes) hostname change:

If you're unsure which pattern your stack uses, open the HEC page in Splunk (Settings → Data Inputs → HTTP Event Collector) and click Global Settingsat the top right. The URL hint is on that screen.

Why: keep Arbiter data isolated from the rest of your Splunk estate so retention, RBAC and dashboards stay tidy. One index per tenant works too if you run several tenants into one Splunk stack.

Top nav → Settings→ Indexes→ New Index:

• Index name: arbiter
• Index Data Type: Events
• App: Search & Reporting
• Leave size and retention on defaults; tune later.

Save. The new index appears in the list immediately.

Why: the token is the credential Arbiter uses to POST events. One token per tenant is cleanest because rotation, revocation and per-tenant volume tracking all key off the token.

a. Enable HEC if it isn't already

Settings → Data Inputs→ HTTP Event Collector. Top right click Global Settings and set All Tokens to Enabled. On Splunk Cloud HEC is already on, this is a no-op.

b. Issue a new token

Back on the HEC page top right New Token:

• Name: arbiter
• Description: Arbiter SIEM egress
• Source name override: leave blank.

Click Next. On the Input Settings page:

• Source type: select New and enter arbiter:security_event (one underscore-free colon-separated string; Splunk treats it as one token).
• App context: leave on Search & Reporting.
• Select Allowed Indexes: tick arbiter.
• Default Index: arbiter.

Click Review → Submit. The token value appears once, formatted like 1a2b3c4d-5e6f-7890-abcd-ef1234567890. Copy it now. If you navigate away without copying, the value is unrecoverable: delete the token and create a new one.

Why: proving the token works from a plain shell before you point Arbiter at it eliminates 90% of the things that can go wrong. If curl can't post, Arbiter can't either.

TOKEN=<paste your HEC token here>
curl -k https://<your-stack>.splunkcloud.com:8088/services/collector/event \
  -H "Authorization: Splunk $TOKEN" \
  -d '{"event":"hello from arbiter","sourcetype":"arbiter:security_event","index":"arbiter"}'

Expected response:

{"text":"Success","code":0}

Then in the Splunk search bar (Apps → Search & Reporting), confirm it landed. Set the time picker to Last 15 minutes:

index=arbiter sourcetype=arbiter:security_event earliest=-15m

You should see the "hello from arbiter" event. If you don't, troubleshoot before moving on:

• DNS error: the http-inputs-prefix may not exist on your stack. Try the URL without the prefix (Step 1, first row).
• TLS verification error: self-signed cert on a self-hosted Splunk. Add -k to curl for the smoke test; in Arbiter, untick Verify TLS certificate on the target config.
• "Invalid token": token wasn't pasted in full. HEC tokens are 36 chars (full UUID); confirm with echo -n "$TOKEN" | wc -c.

Why: the SIEM target row is what tells the Arbiter forwarder where to ship events.

In the Arbiter admin portal, open the tenant you want shipping to Splunk. Switch to the SIEMtab → + Add target.

Save the target, then click the Testpaper-plane icon next to the row. The status flips to OK (green tick) within about 10 seconds on success.

If it stays red, hover the Error icon. The tooltip shows the response from Splunk:

• 401: token missing or wrong.
• 403: token correct but disabled, or not allowed to write to this index.
• 400: payload format issue. Should not happen on this path; if it does, email support.

Why: verify the pipeline end-to-end in two stages before you trust it for production alerting.

a. Canary check (Test button)

In Arbiter, on the SIEM tab, click the paper-plane icon next to the target row. The forwarder posts a single synthetic event matching the stream's schema; the status flips to OK within about 10 seconds.

Confirm it landed in Splunk:

index=arbiter sourcetype=arbiter:security_event earliest=-5m

You should see one event whose body starts "Arbiter SIEM test event...".

b. Real data check

Real events from the tenant start flowing on the next cadence tick. The default for auth_log_rejectsis every 60 seconds, so a failed authentication on your network should be visible within a minute or two:

index=arbiter sourcetype=arbiter:security_event source=arbiter.auth_log_rejects earliest=-15m
| table _time mac_address nas_name auth_method reason auth_policy_name
| sort - _time

If your tenant has any failing auths (blocked MABs, expired certs, default-deny matches), rows appear here. If nothing's appearing, check the Auth log tab in the Arbiter portal first to confirm rejects are actually being produced.

Expected latency

Why: the value of a SIEM over a passive log store is in dashboards and alerts. Starter SPL below.

# Failed-auth rate by hour, useful for a dashboard panel
index=arbiter sourcetype=arbiter:security_event source=arbiter.auth_log_rejects earliest=-7d
| timechart span=1h count

# Top reject reasons over the last 24 hours
index=arbiter sourcetype=arbiter:security_event source=arbiter.auth_log_rejects earliest=-24h
| stats count by reason
| sort - count
| head 10

# Endpoints producing the most rejects (potential misconfig or attack)
index=arbiter sourcetype=arbiter:security_event source=arbiter.auth_log_rejects earliest=-24h
| stats count as attempts, values(reason) as reasons by mac_address
| sort - attempts
| head 20

To turn one of these into an alert, save the search and click Save As → Alert. Set a schedule (e.g. every 5 minutes), a trigger condition (e.g. "Number of Results > 20") and an action (email, webhook, ServiceNow, etc.).

Sample event for context

This is the full shape of an auth-log-reject event as it lands in Splunk:

{
  "event": {
    "time": "2026-05-21T19:39:31.315147Z",
    "event_id": 3637403,
    "tenant_id": 6,
    "mac_address": "b827ebbec2db",
    "user_name": "b827ebbec2db",
    "nas_ip": "192.168.0.21",
    "nas_name": "HOMESW",
    "nas_port": "GigabitEthernet0/1/1",
    "vlan_id": null,
    "ssid": null,
    "ap_mac": null,
    "auth_method": "mab",
    "eap_type": null,
    "reason": "Denied by default policy",
    "policy_name": "Default Deny",
    "auth_policy_name": "Wired MAB",
    "access_profile_name": null,
    "monitor_mode": false,
    "monitor_coerced": false,
    "client_cert_cn": null,
    "client_cert_serial": null,
    "client_cert_issuer": null,
    "client_cert_issuer_org": null,
    "source": "arbiter.auth_log_rejects"
  },
  "time": 1747856371,
  "sourcetype": "arbiter:security_event",
  "source": "arbiter"
}

And a security event:

{
  "event": {
    "time": "2026-05-21T14:30:00.123456Z",
    "event_id": 1042,
    "tenant_id": 6,
    "rule_name": "ssh-on-guest-vlan",
    "severity": "high",
    "mac_address": "aabbccddeeff",
    "nas_ip": "10.0.0.5",
    "nas_port_id": "GigabitEthernet0/1",
    "summary": "SSH (TCP/22) seen from a guest-VLAN endpoint",
    "details": {
      "flow_src": "192.168.30.42",
      "flow_dst_port": 22,
      "flow_proto": "tcp"
    },
    "source": "arbiter.security_events"
  },
  "time": 1747837800,
  "sourcetype": "arbiter:security_event",
  "source": "arbiter"
}