Text Generation Model
Moonshotâs text generation model (referred to as moonshot-v1) is trained to understand both natural and written language. It can generate text output based on the input provided. The input to the model is also known as a âprompt.â We generally recommend that you provide clear instructions and some examples to enable the model to complete the intended task. Designing a prompt is essentially learning how to âtrainâ the model. The moonshot-v1 model can be used for a variety of tasks, including content or code generation, summarization, conversation, and creative writing.Language Model Inference Service
The language model inference service is an API service based on the pre-trained models developed and trained by us (Moonshot AI). In terms of design, we primarily offer a Chat Completions interface externally, which can be used to generate text. However, it does not support access to external resources such as the internet or databases, nor does it support the execution of any code.Token
Text generation models process text in units called Tokens. A Token represents a common sequence of characters. For example, a single English character like âantidisestablishmentarianismâ might be broken down into a combination of several Tokens, while a short and common phrase like âwordâ might be represented by a single Token. Generally speaking, for a typical English text, 1 Token is roughly equivalent to 3-4 English characters. It is important to note that for our text model, the total length of Input and Output cannot exceed the modelâs maximum context length.Rate Limits
How do these rate limits work? Rate limits are measured in four ways: concurrency, RPM (requests per minute), TPM (Tokens per minute), and TPD (Tokens per day). The rate limit can be reached in any of these categories, depending on which one is hit first. For example, you might send 20 requests to ChatCompletions, each with only 100 Tokens, and you would hit the limit (if your RPM limit is 20), even if you havenât reached 200k Tokens in those 20 requests (assuming your TPM limit is 200k). For the gateway, for convenience, we calculate rate limits based on the max_tokens parameter in the request. This means that if your request includes the max_tokens parameter, we will use this parameter to calculate the rate limit. If your request does not include the max_tokens parameter, we will use the default max_tokens parameter to calculate the rate limit. After you make a request, we will determine whether you have reached the rate limit based on the number of Tokens in your request plus the number of max_tokens in your parameter, regardless of the actual number of Tokens generated. In the billing process, we calculate the cost based on the number of Tokens in your request plus the actual number of Tokens generated.Other Important Notes:
- Rate limits are enforced at the user level, not the key level.
- Currently, we share rate limits across all models.