ShopFloor - Jobs & Integrations

Jobs & Integrations

Background processing, scheduled jobs, and external system integration. For wiring and DI setup see 10-architecture.md.

1. Quick map

flowchart LR
    subgraph Inbound[Inbound integrations]
        FB[Fishbowl ERP<br/>MySQL]
        ML[Mercado Libre]
        HP[HP PCB API]
        BR[BitRaser]
        CA[Channel Advisor<br/>temp file]
        PT[Partner systems<br/>via ExternalAPI]
        WTS[Windows Test System]
    end
    subgraph App[CTLShopFloor]
        HF[Hangfire jobs<br/>~18 recurring]
        WAPP[WebAPP]
        EXT[ExternalAPI]
    end
    subgraph Outbound[Outbound integrations]
        BLOB[Azure Blob Storage]
        MONGO[(MongoDB<br/>audit log)]
        SEQ[Seq]
        EMAIL[SMTP<br/>per-project]
        FBOUT[Fishbowl inventory<br/>sync back]
    end
    FB --> HF
    ML --> HF
    BR --> HF
    CA --> HF
    HP --> WAPP
    PT --> EXT
    WTS --> WAPP
    HF --> BLOB
    HF --> EMAIL
    HF --> FBOUT
    WAPP --> MONGO
    WAPP --> SEQ

2. Hangfire configuration

Aspect	Detail
Backing store	SQL Server (same connection family as the main DB)
Dashboard	`/hangfire` (authenticated) — only registered when `JOBS:allowJobsExecution=true`
Registration	`ServiceExtensions.ConfigureBackgroundJobs` in WebAPP
Gating	All recurring-job registrations skipped if `JOBS:allowJobsExecution=false`. This is the prod-safety mechanism — dev machines should never have it set to true.
Error notifications	`BaseJob.EmailDevelopersOnError()` (⚠️ wrapped in bare `catch {}` so failures may be silent)

3. Job inventory

~18 active recurring jobs + a few disabled / stub jobs. Cron times are UTC; CST offset noted where the comment in code calls it out.

Active recurring jobs

Job	Purpose	Schedule (UTC)	Touches
`NotifyUsersOfProductEOLJob`	Emails owners when products reach EOL in 30/60/90 days. Pulls from `ProductEOL`.	Daily	Email via `NotificationRecipient` (Notifications.ProductEOL)
`SyncSalesAndInventoryWithWorkTrackingJob`	Bidirectional inventory sync for the two hardcoded tenants.	`30 11,22,13-23/1 * * *` (6:30 AM, 5:30 PM, hourly 8:30 AM–4:30 PM CST)	Fishbowl ↔ WorkTracking. ⚠️ Hardcoded tenant IDs.
`PurgeErrorLogJob`	Deletes aged rows from the Serilog `ErrorLog` table.	Daily at 10:00	None
`TranslateSystemInformationEntriesJob`	Processes WorkTracking rows missing `SystemInformation`; maps from `EngineeringData`.	`0 1-3,14-23/1 * * 1-5` (7 PM–9 PM + 8 AM–5 PM CST, weekdays)	Internal
`AssignSystemVarianceCodeToNewUnitsJob`	Assigns `SystemInformationVarianceCode` to units lacking one.	Same as above	Internal
`GenerateAndNotifyOfHpCoaReportJob`	Monthly CSV of DPKs injected last month, emailed to recipients.	`30 14 1 * *` (8:30 AM CST, 1st of month)	HP COA (DPK) data, email. ⚠️ Hardcoded tenant ID `36650844-da39-4b72-bf8f-ec989db49a27`.
`WorkOrderAcquisitionCostJob`	Monthly: reports work orders where acquisition cost wasn’t fully assigned.	`30 13 1 * *` (7:30 AM CST, 1st of month)	Email via `Notifications.WorkOrderAcquisitionCost`
`BitraserLogDownloadJob`	Downloads drive erase + verification logs from BitRaser API; persists to `DriveEraseLog` / `DriveVerificationLog`.	Daily at 3:00 (3:00 AM CST)	BitRaser API
`GenerateTrackingReportWithSixMonths`	Daily 6-month tracking Excel report; uploads to Azure Blob `trackingreport{tenantname}`; deletes blobs older than 14 days.	Daily at 9:00 (1:00 AM CST)	Azure Blob Storage
`FishbowlSalesImportJob`	Imports prior-day fulfilled/shipped sales orders from Fishbowl into `ProductSalesOrder` + line items.	Daily at 7:30 (1:30 AM CST)	Fishbowl API
`MercadoLibreSalesImportJob`	Imports prior-day orders from Mercado Libre marketplace.	Daily at 8:00 (2:00 AM CST)	Mercado Libre API
`UpdateSalesChannelJob`	Cross-references `SalesImport` to `ProductSalesOrder.ChannelName` via `ChannelOrderID` and the CA temp import file.	Daily at 8:30 (2:30 AM CST)	`SalesImport` table
`SalesInventoryUpdateJob`	Matches sales rows (Fishbowl + Mercado Libre) to WorkTracking units by serial (FIFO); sets `UnitSoldDate` / `UnitShippedDate` / station; links `SalesImportLineItemDetail.WorkTrackingID`. ⚠️ Hardcoded facility → tenant mapping (`MX/CA → Mexicali, SL → Dallas SL`).	Daily at 9:00 (3:00 AM CST)	Fishbowl + Mercado Libre + WorkTracking

Manually-invokable jobs

Job	Purpose	Trigger
`OemProductDownloadJob`	Fetch OEM product data by SKU.	Manual (no cron)

Stub / disabled / commented-out jobs

Job	Status	Reason
`ConvertSalesImportJob`	Stub	Mercado Libre → `ProductSalesOrder` conversion incomplete (commented-out body); currency / bundle handling deferred
`TranslateEngineeringDataEntriesJob`	Stub	Converts `EngineeringData` rows → `SystemInformation`; mapper logic incomplete; sets `SysInfoConversionFailed` on error
`PurgeAuditLogJob`	Disabled	Comment at line 47: `"REMOVED IN FAVOR OF DIRECT INSERT TO MONGO"`. SQL `AuditLog` table retention now handled by Mongo TTL (⚠️ confirm SQL table truly receives no writes).

4. Anti-patterns in jobs

Consolidated from inspection of WebAPP/Jobs/*:

Smell	Where	Risk
Bare `catch {}`	`AssignSystemVarianceCodeToNewUnitsJob:36`, `TranslateSystemInformationEntriesJob:36`, `BaseJob.EmailDevelopersOnError():60`	Silent failures — operations may report success while doing nothing
Hardcoded tenant IDs	`SyncSalesAndInventoryWithWorkTrackingJob:14`, `GenerateAndNotifyOfHpCoaReportJob:56` (TODO comments flag)	New tenant onboarding requires code change
Hardcoded facility → tenant mapping	`SalesInventoryUpdateJob:115–126`	Same as above; fragile string matching (`MX`, `CA`, `SL`)
No idempotency gating	`FishbowlSalesImportJob`	Relies on external ref dedup; concurrent runs could double-import
No retry / circuit breaker	All external API calls (BitRaser, Fishbowl, Mercado Libre, HP PCB)	Single API blip = job fails for the day
Per-order swallow	`FishbowlSalesImportJob:123`	Errors on individual orders logged but not retried
Silent skip on missing data	`SalesInventoryUpdateJob.UpdateLineItemDetailAndWorkTracking:210`	Returns when WorkTracking not found; no surfacing
Incomplete stubs in production codebase	`ConvertSalesImportJob`, `TranslateEngineeringDataEntriesJob`	Confusing — appear to be wired but do nothing useful

5. External integrations

Fishbowl (MySQL ERP)

Direction: primarily inbound (read) with one outbound sync.
Inbound: sales orders, sales-order line items, sales-order line-item details (serial numbers + facility/location), parts inventory.
Outbound: the always-on SyncSalesAndInventoryWithWorkTrackingJob writes inventory updates back to Fishbowl for the two hardcoded tenants.
Wiring: FishbowlDbContext (Pomelo MySQL, NoTracking globally), FishBowlInventoryIntegrationService, IFishbowlService (contracts).
Cadence: see job table above.

Mercado Libre

Direction: inbound only.
Inbound: marketplace orders.
Wiring: IMercadoLibreService (Services.WebAppServicesImplementation.Domain).
Cadence: daily.

HP PCB API

Direction: inbound, read-only.
What it does: product autocomplete, catalog, product info, tech specs, images by HP model.
Wiring: Services.HPPCBApi.HPPCBApiService, named HttpClient "pcb".
Trigger: ad-hoc (from OemProductService, OEM lookup flows).

BitRaser

Direction: inbound, read-only.
What it does: retrieves drive-erasure and verification reports for the past day.
Wiring: IBitraserClient.
Persistence: DriveEraseLog, DriveVerificationLog tables + JSON copy in Azure Blob.

Channel Advisor temp import file

Direction: inbound, file-based.
What it does: provides ChannelName enrichment to reconcile against SalesImport.SiteOrderID.
Wiring: UpdateSalesChannelJob reads the file and updates ProductSalesOrder.ChannelName.

Partner systems (ExternalAPI)

Direction: inbound, push.
What partners do: submit inbound orders, query shipment status.
Wiring: CTLShopFloor.Presentation.ExternalAPI with ApiKeyAuthMiddleware + BearerTokenAuthMiddleware; custom ExternalApiMultiTenantStrategy.
Models: CreateNewInboundOrderRequest, CreateNewOutboundOrderRequest.
Controllers: AuthController, InboundOrderController, TestController.

Windows Test System

Direction: bidirectional (offline).
Inbound: unit physically moves there from the Image Download station.
Outbound: uploads WindowsTestResultsUpload, populates SystemInformation + SystemInformationDisk + SystemInformationGraphic + 8 Engineering* tables; sets WorkTracking.SystemInformationBlob and WindowsTestResultBlob; injects DigitalProductKey and sets AssignedSerialNumber + InjectionTime.

Azure Blob Storage

Direction: outbound (writes) + inbound (reads for downloads).
Containers observed:
- trackingreport{tenantname} — 6-month tracking reports (daily, Excel).
- blobstorage (data-protection keys for ASP.NET Identity).
- Drive erasure JSON dumps (BitRaser job, lines 75, 127).
- Image / PDF uploads — managed via AzureBlobContainersControllerService + AzureBlobStorageContainers entity.
Cleanup: the tracking-report job deletes blobs older than 14 days.

MongoDB (audit)

Direction: outbound.
What: every entity change via Audit.NET interceptor.
Database: auditLog
Collection: shopFloor
Ignored entities: AuditLog, SalesImport
Ignored fields: WorkTracking.SystemInformationBlob, WorkTracking.WindowsTestResultBlob (too large)
Note: the SQL AuditLog table still exists; ⚠️ confirm no writes still hit it.

Seq (centralized logging)

Direction: outbound.
What: all Serilog events at minimum level Verbose.
Used by: every Serilog-enabled component.

6. Email

Two parallel email pipelines:

Transactional (interactive)

IEmailSender → EmailSender (custom implementation, registered Scoped in Program.cs line 55).
Per-project SMTP via EmailServerSettings: SMTP_Server, Port, FromAddress, UserName (encrypted password), Security (TLS/SSL), Active flag.
Sending is gated by EmailServerSettings.Active per project.

Alert / job notification

Notification enum (13 types) defines alert categories.
NotificationRecipient rows hold destination emails per notification type.
Jobs query notificationRecipientRepository.FindByCondition(r => r.NotificationID == (int)Notifications.X) and email each Destination.

Notification types observed:

ID	Name	Source
1	`ProductEOL`	`NotifyUsersOfProductEOLJob`
2	`EnrollmentTestResults`	⚠️ source not located
3	`DpkInventoryReport`	⚠️ source not located
4	`UnitsByStationType`	⚠️ source not located
5	`HpCoaReport`	`GenerateAndNotifyOfHpCoaReportJob`
11	`WorkOrderAcquisitionCost`	`WorkOrderAcquisitionCostJob`
12	`NewModelBOMHeader`	RMA out-of-system path when a new model is auto-created
13	`DeveloperAlert`	`BaseJob.EmailDevelopersOnError()`

(Other enum values exist but no observed sources — possible dead code.)

Per-tenant SMTP

EmailServerSettings.ProjectID plus Active filter means each project can have its own SMTP. The BaseJob and BitraserLogDownloadJob (line 147) both look up SMTP per-tenant.

7. Notifications (in-app)

⚠️ No in-app notification system observed. Notification + NotificationRecipient are an email-distribution-list mechanism, not a UI notification queue. Confirm before designing a UI notification feature on top of them.

8. PDF generation

PDFControllerService exists with methods for fetching pallets (internal / transfer / box / inbound) — likely for label and slip generation. Exact PDF rendering logic was not fully traced. Likely outputs:

Pallet labels (barcode + contents)
Bill of Lading slips (WorkOrders.BOLFileIdentifier linkage)
Inbound / outbound packing slips

⚠️ Confirm whether PDF rendering uses Syncfusion or a separate library.

9. File uploads

WebAPP request size limit: 1 GB (Program.cs lines 18, 57–69) — generous, used for pre-alert Excel and image uploads.
Pre-alert Excel parsing: RmaForOutOfSystemUnitsControllerService.ProcessFile (lines 279–386) — also auto-creates BOMHeader rows for unknown SKUs and emails admins.
General file upload mapping: FileUploadMappingService.

10. SalesImport pipeline (end-to-end)

This is the most complex job chain. Documenting it explicitly because the order matters.

sequenceDiagram
    participant FB as Fishbowl
    participant ML as Mercado Libre
    participant CA as Channel Advisor temp file
    participant DB as Database
    participant J1 as FishbowlSalesImportJob<br/>1:30 AM CST
    participant J2 as MercadoLibreSalesImportJob<br/>2:00 AM CST
    participant J3 as UpdateSalesChannelJob<br/>2:30 AM CST
    participant J4 as SalesInventoryUpdateJob<br/>3:00 AM CST
    FB->>J1: prior-day orders
    J1->>DB: ProductSalesOrder + line items + details
    ML->>J2: prior-day orders
    J2->>DB: ProductSalesOrder + line items + details
    CA->>J3: temp file
    J3->>DB: ProductSalesOrder.ChannelName updated
    DB->>J4: SalesImport rows
    J4->>DB: Match by serial → WorkTracking.UnitSoldDate / UnitShippedDate / SentToSalesDate; SalesImportLineItemDetail.WorkTrackingID

After this chain runs, every sold unit’s WorkTracking row should reflect its disposition. If WorkTracking was not found for a serial, J4 logs and continues (line 210) — these go to a “couldn’t reconcile” silent failure.

🔧 Refactor finding: silent failures in the sales pipeline are an audit gap. There should at minimum be a SalesImportReconciliationFailures table or alert.

11. ExternalAPI vs. WebAPI

Two distinct API surfaces:

Aspect	`Presentation.ExternalAPI`	`Presentation.WebAPI`
Audience	Partner systems	Internal admin, imaging stations, OEM lookup
Auth	API Key + Bearer	API Key only
Tenant resolution	Custom `ExternalApiMultiTenantStrategy`	Standard header / session / route
Controllers	`AuthController`, `InboundOrderController`, `TestController`	`BomHeaderController`, `DigitalProductKeysController`, `ImagingStationController`, `LoginController`, `OemProductController`, `ProjectsController`, `TenantsController`, `UserController`, …
Purpose	Push inbound orders, query shipment status	Internal data access, admin ops, imaging-station data

12. Configuration touch-points

Config flag	Effect on jobs/integrations
`JOBS:allowJobsExecution`	Gates Hangfire registration. Must be false on dev machines.
`Migrations:allowMigrations`	Gates EF migration application. Independent of jobs.
`ConnectionStrings:RepositoryDb`	Primary SQL Server.
`ConnectionStrings:TenantDb`	Tenant registry SQL Server.
`ConnectionStrings:FishbowlDb`	Fishbowl MySQL.
`ConnectionStrings:AuditLog` (or similar)	MongoDB for audit.
`ConnectionStrings:blobstorage`	Azure Blob.
`AppSettings:Token`	JWT signing secret. ⚠️ Currently in `appsettings.json` — see security findings in 10-architecture.md §7.
`SlowQueryThreshold`	EF Core slow-query log threshold (default 5000 ms).

13. Reading on from here

For what the units these jobs touch are doing → 02-unit-flow.md
For what the SalesImport / Fishbowl entities look like → 11-data-model.md §14 Cluster L
For the consolidated refactor list → 20-refactor-findings.md