RecommendVmSkuTokenBudget interface
The per-request token budget used to size the inference key-value cache.
Properties
| max |
The maximum number of generated output tokens per inference request. Must be supplied together with |
| prompt |
The number of prompt, or input, tokens per inference request. Must be supplied together with |
Property Details
maxOutputTokens
The maximum number of generated output tokens per inference request. Must be supplied together with promptTokens, and the two must total from 1 through 131072.
maxOutputTokens?: numberProperty Value
number
promptTokens
The number of prompt, or input, tokens per inference request. Must be supplied together with maxOutputTokens, and the two must total from 1 through 131072.
promptTokens?: numberProperty Value
number
