A workflow engine like Temporal (or DBOS, Conductor, Restate) keeps long-running work alive across crashes, restarts, and partial failures.
Temporal does that by running a Workflow Execution and persisting an Event History, an append-only record similar to a Commit Log. The Event History records facts such as a workflow starting, an Activity being scheduled, an Activity completing, a timer firing, a signal arriving.
Workflows vs Activities
→ A Workflow defines the overall flow of the application by organizing a series of activities and are carried out by running that code, which results in an execution.
→ An Activity executes a single, well-defined unit of work, such as: a database write, an API call, or sending an email.
Booking Workflow
An example workflow that retries when the payment activity fails
Orchestration
A Temporal application orchestrates many components. The Temporal Service stores history, schedules tasks, and coordinates state transitions. Workers poll Task Queues, execute Workflow and Activity code, and report results back to the service. When a Worker resumes a Workflow, it replays the Event History to rebuild the workflow state. Completed activites won't be recomputed because their results are serialized and stored in ActivityTaskCompleted in the Event History.
The coordination loop
The Temporal Service queues pending tasks; a Worker processes the queue, running each activity and reporting every outcome to the Event History.
Retries and idempotency
Durable execution records progress, but it does not make side effects exactly once. If a Worker sends an email and then crashes before completion is recorded, a retry may run that code again. Temporal retries Activities by default. It's recommended that Activities are idempotent: use idempotency keys, unique constraints, guarded updates, or similar techniques to avoid performing a side effect more than once.
Design for Idempotency
When the Worker crashes after sending an email and Temporal retries, a stable idempotency key lets the provider recognize the retry and skip the resend, while sending no key at all leaves it nothing to match on, so a duplicate goes out.
- Send first email
- Crash before ack
- Temporal retries
- Provider responds
Worker crash mechanics
If a Worker dies during a Workflow Task, another Worker can replay the Event History and continue the workflow execution from the same point. If it dies during an Activity Task, Temporal cannot know whether the external side effect happened; it waits for a timeout, then retries if the Activity's policy allows it.
Worker Failure
When a Worker dies mid-task, a Workflow Task records its whole recovery in Event History, but an Activity leaves no trace at all, which is why it has to be idempotent.
At LinkedIn
My team, LBP Checkout & Ordering, uses Temporal to orchestrate Order placement from the checkout page. When someone clicks the Place Order button, we start a workflow that waits for payment to succed, then proceed to fulfill their order by provisioning a subscription.
Temporal is useful here because of its ability to survive failures; steps in the workflow can fail, but will eventually succeed. For example, the provisioning service may be down temporarily due to an incident. When it is available again, the workflow can proceed and fulfill a customer's order, allowing them access to the subscription they purchased.