89 lines
3.2 KiB
Markdown
89 lines
3.2 KiB
Markdown
# yksa-source-poller
|
|
|
|
The machine that polls upstreams on a schedule: a source model, a run log, the
|
|
dispatcher and the bookkeeping around one attempt. The cargo — what a fetched
|
|
record *is* and what happens to it — stays in the service.
|
|
|
|
Extracted from `tdas/yksa_tdas/sources/` and `tle/yksa_tle/sources/`, which were
|
|
the same ~500 lines with different payload types.
|
|
|
|
## Install
|
|
|
|
```
|
|
yksa-source-poller @ git+https://git.intra.yksa.space/web/yksa-source-poller.git@v0.1.0
|
|
```
|
|
|
|
Requires [yksa-django-kit](../yksa-django-kit) for `RunStatus` and `read_secret`.
|
|
It is **not** added to `INSTALLED_APPS`: every model in it is abstract, so it
|
|
ships no tables and needs no app config.
|
|
|
|
## Use
|
|
|
|
```python
|
|
# sources/models.py
|
|
class TelemetrySource(AbstractPollingSource):
|
|
SUBSCRIPTION_ID_FIELDS = {"norad": "norad_cat_id", "satnogs": "satnogs_id"}
|
|
|
|
kind = models.CharField(max_length=32, choices=SourceKind.choices)
|
|
subscription_id_kind = models.CharField(..., default=SubscriptionIdKind.SATNOGS)
|
|
tracked_satellites = models.ManyToManyField("satellites.Satellite", blank=True)
|
|
|
|
|
|
class SourceRun(AbstractSourceRun):
|
|
source = models.ForeignKey(TelemetrySource, on_delete=models.CASCADE,
|
|
related_name="runs")
|
|
```
|
|
|
|
```python
|
|
# sources/tasks.py
|
|
POLLER = Poller(
|
|
source_model=TelemetrySource,
|
|
run_model=SourceRun,
|
|
adapters=ADAPTERS,
|
|
persist=_persist_frame, # -> True when the record is new
|
|
track=_track_latest_rx, # optional: extra source fields on success
|
|
unpollable_kinds=(SourceKind.INTERNAL_PUSH,),
|
|
)
|
|
|
|
|
|
@shared_task(bind=True, max_retries=3, default_retry_delay=120)
|
|
def run_source_task(self, source_id):
|
|
try:
|
|
return run_source(POLLER, source_id)
|
|
except Exception as exc:
|
|
raise self.retry(exc=exc)
|
|
```
|
|
|
|
The Celery tasks stay in the service on purpose: `CELERY_BEAT_SCHEDULE` names
|
|
them by dotted path, and a task that moved into this package would silently stop
|
|
being scheduled.
|
|
|
|
## Why the models are abstract
|
|
|
|
The two concrete source models differ where they have to — which satellite model
|
|
the M2M points at, which identifiers the upstream accepts, and (in tdas) a
|
|
`last_rx_at` watermark. Everything else was identical. Abstract bases share the
|
|
identical part without pretending the rest is the same.
|
|
|
|
Field definitions here reproduce what the services already had, so adopting the
|
|
package needs **no migration**. Where a service's own definition differed — a
|
|
different `poll_interval_sec` default, different wording — the service overrides
|
|
the field, which Django allows for abstract inheritance. Check with
|
|
`manage.py makemigrations --check --dry-run` after wiring it up.
|
|
|
|
## What the runner guarantees
|
|
|
|
Every exit path — success, failure, unconfigured adapter, missing adapter —
|
|
writes `last_status`, updates a counter and re-arms `next_poll_at`. That is the
|
|
reason to share it: a path that forgets leaves the dispatcher either hammering
|
|
the source once a minute or ignoring it forever, and both shipped once.
|
|
|
|
`NotImplementedError` from an adapter is a *skip*, not a failure: a half-configured
|
|
source must not look like an outage.
|
|
|
|
## Tests
|
|
|
|
```sh
|
|
pip install -e ".[test]" ../yksa-django-kit
|
|
python -m pytest
|
|
```
|