Summary
- Streaming responses require robust SSE parsing to handle incomplete chunks and final token usage data.
- JSON mode failures usually stem from the model generating trailing text outside the strict JSON schema.
- The 100k context window is shared between prompt and completion, so long outputs reduce available input space.
- Cost estimation must account for the higher price of output tokens compared to input tokens.
Why Your Streaming Response Is Hanging
Streaming responses hang when the client fails to read the final Server-Sent Event (SSE) chunk containing token usage data. Our API streams tokens as they are generated, but the connection only fully closes after the last chunk arrives. If your client code disconnects early or ignores the final chunk, you lose the exact token counts needed for billing verification.
Read every chunk until the stream ends. Do not assume the last chunk contains only the final token. The final chunk in our SSE stream includes the usage field with input and output token totals. If you discard this data, you cannot accurately track your prepaid credit consumption.
Common Pitfalls
- Ignoring the final SSE event.
- Timeouts set too low for long completions.
- Not handling partial JSON in streaming mode.
Always consume the entire stream to ensure the server knows the client received the full response. This prevents resource leaks on our side and ensures you have the data needed for your own logs.
Handling JSON Mode Errors in Production
JSON mode errors occur because the model may generate text after the closing brace, violating strict JSON parsing rules. When you set response_format to {"type": "json_object", the model attempts to output valid JSON, but it often adds conversational filler like 'Here is the data:' after the closing bracket. Your parser then crashes on the extra text.
Strip trailing text after the last valid closing brace. Do not rely on the model to stop exactly at the end of the JSON structure. You must implement a post-processing step to find the last '}' or ']' and discard everything after it. This is critical when parsing arrays or nested objects.
Best Practices
- Use a regex to extract the last valid JSON block.
- Validate the extracted string with a JSON parser.
- Retry with a lower temperature if the output is chaotic.
This approach ensures your production pipeline remains stable even when the model decides to be chatty. Uncensored models are less constrained, so they may feel more 'creative' with their formatting.
Understanding Context Window Limits
The context window limit is 100,000 tokens for the combined prompt and completion. This means if your input is 50,000 tokens, you can only generate up to 50,000 tokens before hitting the limit. The limit is not just for input; it is a shared budget for the entire request.
Calculate your available output space by subtracting the input token count from 100,000. If you send a large system prompt or a long history of messages, your available output space shrinks accordingly. This is crucial for applications that maintain long conversation histories.
Calculation Example
- Total Window: 100,000 tokens.
- Input Tokens: 20,000 tokens.
- Max Output: 80,000 tokens.
If you exceed this limit, the API returns an error. Monitor your token usage closely to avoid unexpected truncation of your model's response. The model does not automatically truncate; it simply refuses the request if the total exceeds the window.
Token Counting and Cost Estimation
Cost estimation must account for the price difference between input and output tokens. Input tokens cost $0.15 per 1M tokens, while output tokens cost $0.50 per 1M tokens. Output tokens are significantly more expensive, so generating long responses will drive up your costs faster than sending long prompts.
Estimate your costs by multiplying the expected token count by the respective price. For example, 10,000 input tokens and 5,000 output tokens would cost: (10,000/1,000,000 * $0.15) + (5,000/1,000,000 * $0.50). This equals $0.0015 + $0.0025 = $0.004 per request.
Pricing Table
| Token Type | Price per 1M Tokens |
|---|---|
| Input | $0.15 |
| Output | $0.50 |
Track your usage via the final SSE chunk or the API response. This ensures you can budget accurately for high-volume applications. Prepaid credit never expires, so you can plan your spending over time without monthly fees.
Rate Limiting: 300 RPM Explained
Rate limiting caps your requests at 300 per minute per API key. This limit applies to all requests, regardless of payload size or token count. If you exceed this limit, the API returns a 429 error. You must implement exponential backoff to handle these errors gracefully.
Monitor your request rate and adjust your concurrency accordingly. If you are using multiple threads or processes, ensure they share the same API key and collectively stay under the 300 RPM limit. Exceeding this limit will result in temporary errors until the rate resets.
Handling 429 Errors
- Implement exponential backoff.
- Log failed requests for analysis.
- Reduce concurrency if limits are frequent.
Our API is designed for high-volume use, but the rate limit ensures fair usage across all clients. If you need higher limits, you can generate a new API key, but note that each account is limited to one active key at a time.
Troubleshooting Function Calling Failures
Function calling failures occur when the model's tool definition does not match the expected schema or when the model fails to output valid JSON for the tool arguments. Ensure your tool definitions are precise and include all required parameters. Ambiguous schemas lead to malformed tool calls.
Validate the tool arguments before executing the function. The model may generate arguments that are missing required fields or contain incorrect types. Your client code should handle these errors by retrying with a clearer prompt or adjusting the tool definition.
Debugging Steps
- Check tool definition syntax.
- Validate argument types and values.
- Review model output for missing fields.
Function calling is a powerful feature, but it requires careful schema design. Uncensored models may be less consistent with tool definitions than their filtered counterparts, so rigorous validation is essential.
Why 'uncensored' Isn't Totally Unlimited
The 'uncensored' label means the model does not refuse lawful adult, fictional, or controversial topics, but it still has a hard limit on sexual content involving minors. This is the only hard content restriction that always applies. Requests containing such content will be refused, even if the rest of the prompt is perfectly valid.
Other refusals may still occur due to technical errors or rate limits. The model is not magic; it can still fail to generate a response if the input is too complex or the context window is exceeded. Understand that 'uncensored' refers to content policy, not technical reliability.
Content Limits
- No sexual content involving minors.
- Lawful adult content is allowed.
- Technical errors can still cause refusals.
This distinction is important for applications that need to handle a wide range of topics. Knowing the specific limit helps you design your filtering logic correctly.
Debugging 400 vs 429 Errors
400 errors indicate a problem with the request itself, such as invalid JSON, missing required fields, or exceeding the 8 MB request body limit. 429 errors indicate that you have exceeded the rate limit of 300 requests per minute. Distinguishing between these errors helps you fix the right issue quickly.
For 400 errors, check your request payload and schema. For 429 errors, implement backoff and reduce concurrency. If you receive a 400 error with a message about the request body size, ensure you are not sending too much data in a single request. The 8 MB limit is strict.
Error Codes
| Error Code | Meaning |
|---|---|
| 400 | Bad Request |
| 429 | Rate Limit Exceeded |
Logging these errors with their context (payload, timestamp) makes debugging much easier. Use this data to optimize your integration and avoid repeated failures.
Next Steps for Heretic LLM Optimization
Optimization begins with monitoring your token usage and error rates. Use the token data from the final SSE chunk to track your actual consumption. Compare this against your estimated costs to ensure accuracy. Adjust your concurrency and rate limiting strategies based on your observed usage patterns.
Consider using a lower temperature for more consistent JSON output or function calling. If you need more creative responses, increase the temperature but be prepared for more parsing errors. Experiment with different parameters to find the right balance for your use case.
Optimization Tips
- Monitor token usage closely.
- Adjust temperature for consistency vs. creativity.
- Implement robust error handling.
Start with the free trial to test your integration. This is a low-risk way to verify that your client code handles streaming, JSON mode, and rate limits correctly before committing to prepaid credit.
