Endpoint Consumers and Custom Integration

Push or pull?#

Before configuring a consumer, consider whether the destination should fetch instead.

Infohub pushes — a consumer sends when values change. Good when the destination wants data promptly, has an endpoint that accepts it, and cannot conveniently query.

The destination pulls — it reads from Infohub on its own schedule. Good when the destination already knows how to query, when it wants control over timing, or when it needs to re-fetch history.

Pull is often the better answer and is frequently overlooked. A destination that can query needs no consumer at all, no credential held by Infohub, no retry logic, and no coordination when it goes offline for maintenance. It also means the destination's owner controls their own integration, which usually means it stays working.

Push is the right choice when the destination genuinely cannot pull, when latency matters, or when the destination is a stream or broker by design.

The Endpoint Consumer#

The general-purpose outbound consumer. It posts to a URL or writes to a file, with a body you define.

Body Text is where the work is. It is a template, so you shape the payload to whatever the destination expects rather than being given a fixed format.

Building the body#

Work from the destination's actual specification, not from an assumption.

1. Get an example of a request the destination accepts, from its documentation or its team.
2. Send it manually with a generic HTTP client and confirm it is accepted. This proves the endpoint, the authentication, and the shape before Infohub is involved at all.
3. Build the Body Text to reproduce that shape.
4. Test with one measure, and confirm at the destination that it arrived correctly.
5. Then scale up.

Step 2 is the one people skip, and it is the one that separates a half-hour job from a half-day one. If you cannot get the endpoint to accept a hand-crafted request, Infohub will not do better.

Be precise about timestamps#

Whatever the body contains, make the timestamp unambiguous:

• State the units if publishing epoch values. Seconds and milliseconds are indistinguishable to a reader, and getting it wrong puts values decades away.
• Publish UTC, or include an explicit offset. A timestamp with no offset will be resolved differently by every consumer of it.

This is the same problem the Web Service data source faces reading other people's payloads. Having been on the receiving end is a good argument for being precise when publishing.

Volume and the destination's tolerance#

An endpoint that comfortably handles a message a minute may not handle a backfill.

Infohub republishes restated values, so a correction to a year of history produces a year of requests in a burst. Against an internal API with no rate limiting, that may simply be slow. Against a metered third-party service, it may be expensive or rejected.

Defences:

• Keep the published set narrow with the namespace filter.
• Publish at the interval the destination needs.
• Warn the destination's owner before a large backfill, or exclude backfilled periods.
• Set a realistic Timeout so a slow endpoint fails rather than holding things up.

File output#

Where the destination collects files rather than accepting requests, the file options handle it.

The recurring gotcha: the Integration Engine runs as the Flow Bootstrap service account, so that account needs write access to the destination. A path that works from your workstation proves nothing. Use a UNC path, not a mapped drive letter — mappings are per user and the service will not have them.

Agree a naming and rotation convention with whoever collects the files, or the folder becomes unmanageable within months.

Designing an integration that survives#

The technical configuration is the easy part. Integrations fail over time for predictable, non-technical reasons.

Write down the contract. What is sent, at what interval, in what format, identified how, with timestamps in what units and what zone. One page. Both sides keep a copy. This document is what makes the integration diagnosable in two years.

Name an owner on each side. An integration with no owner degrades silently. When it breaks in eight months, somebody must be called.

Record the credentials and their expiry. Every API key, certificate, and password has a lifetime. An integration that has run untouched for eighteen months will stop on a date somebody could have known.

Decide what happens when the destination is down. Does Infohub retry? For how long? Is a gap acceptable, or must the data be recoverable? Answer this before it happens.

Monitor from the destination's side. Infohub can tell you it sent successfully. Only the destination can tell you the data arrived and is being used. Ask its owner to alert on staleness.

Version the payload. If the body shape may ever change, include a version field from the start. It costs nothing now and it is what lets you change the format later without breaking every consumer simultaneously.

When to write something else instead#

Infohub's consumers cover a lot, and sometimes the right answer is outside Infohub.

A read-only view on the Infohub database. For a destination that can query SQL and wants full flexibility, a view exposing the measures and periods it needs is often simpler than any consumer, and it puts the destination in control of its own timing and retries. Coordinate this with whoever owns the Infohub database, and keep it read-only.

A small intermediary service. Where a destination needs data reshaped in a way no template can express — aggregated differently, joined with another source, transformed — a service that reads from Infohub and writes to the destination is more maintainable than a heroic Body Text template.

Neither is a failure of Infohub. Choosing the right integration boundary matters more than doing everything inside one tool.

Diagnosing#

See also#

• What a Data Consumer Is
• Consumer Reference
• Kafka, Kinesis, and Event Hub
• Web Service Data Sources