Skip to content

Commit

Permalink
check resource file size max size
Browse files Browse the repository at this point in the history
  • Loading branch information
@bowenliang123
bowenliang123 committed Oct 18, 2024
1 parent f8606f4 commit 287c0dc
Show file tree
Hide file tree
Showing 3 changed files with 42 additions and 8 deletions.
14 changes: 8 additions & 6 deletions docs/configuration/settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,14 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

### Batch

| Key | Default | Meaning | Type | Since |
|---------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.batch.application.check.interval | PT5S | The interval to check batch job application information. | duration | 1.6.0 |
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |
| Key | Default | Meaning | Type | Since |
|---------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
| kyuubi.batch.application.check.interval | PT5S | The interval to check batch job application information. | duration | 1.6.0 |
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
| kyuubi.batch.extra.resource.file.max.size | 104857600 | The maximum size in bytes of each extra resource file that can be uploaded in a batch creating request via REST API. 0 or negative value means no limit. | long | 1.10.0 |
| kyuubi.batch.resource.file.max.size | 104857600 | The maximum size in bytes of the resource file that can be uploaded in a batch creating request via REST API. 0 or negative value means no limit. | long | 1.10.0 |
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |

### Credentials

Expand Down
20 changes: 20 additions & 0 deletions kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1876,6 +1876,26 @@ object KyuubiConf {
.booleanConf
.createWithDefault(true)

val BATCH_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
buildConf("kyuubi.batch.resource.file.max.size")
.doc("The maximum size in bytes of the resource file" +
" that can be uploaded in a batch creating request via REST API." +
" 0 or negative value means no limit.")
.version("1.10.0")
.serverOnly
.longConf
.createWithDefault(100 * 1024 * 1024)

val BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
buildConf("kyuubi.batch.extra.resource.file.max.size")
.doc("The maximum size in bytes of each extra resource file" +
" that can be uploaded in a batch creating request via REST API." +
" 0 or negative value means no limit.")
.version("1.10.0")
.serverOnly
.longConf
.createWithDefault(100 * 1024 * 1024)

val BATCH_SUBMITTER_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.batch.submitter.enabled")
.internal
Expand Down
16 changes: 14 additions & 2 deletions kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.kyuubi.server.api.v1

import java.io.InputStream
import java.nio.file.{Path => JPath}
import java.nio.file.{Files, Path => JPath}
import java.util
import java.util.{Collections, Locale, UUID}
import java.util.concurrent.ConcurrentHashMap
Expand All @@ -42,7 +42,7 @@ import org.apache.kyuubi.client.exception.KyuubiRestException
import org.apache.kyuubi.client.util.BatchUtils._
import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.config.KyuubiReservedKeys._
import org.apache.kyuubi.engine.{ApplicationInfo, ApplicationManagerInfo, ApplicationState, KillResponse, KyuubiApplicationManager}
import org.apache.kyuubi.engine._
import org.apache.kyuubi.operation.{BatchJobSubmission, FetchOrientation, OperationState}
import org.apache.kyuubi.server.api.ApiRequestContext
import org.apache.kyuubi.server.api.v1.BatchesResource._
Expand All @@ -65,6 +65,8 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
fe.getConf.get(BATCH_INTERNAL_REST_CLIENT_REQUEST_ATTEMPT_WAIT).toInt
private lazy val internalSecurityEnabled =
fe.getConf.get(ENGINE_SECURITY_ENABLED)
private lazy val resourceFileMaxSize = fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
private lazy val extraResourceFileMaxSize = fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)

private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
Expand Down Expand Up @@ -585,6 +587,10 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
uploadFileFolderPath: JPath): Unit = {
try {
val tempFile = Utils.writeToTempFile(inputStream, uploadFileFolderPath, fileName)
if (resourceFileMaxSize > 0 && Files.size(tempFile.toPath) > resourceFileMaxSize) {
throw new RuntimeException(
s"Resource file $fileName exceeds the maximum size limit $resourceFileMaxSize bytes")
}
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
request.setResource(tempFile.getPath)
} catch {
Expand Down Expand Up @@ -621,6 +627,12 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
filePart.getValueAs(classOf[InputStream]),
uploadFileFolderPath,
fileName)
if (extraResourceFileMaxSize > 0
&& Files.size(tempFile.toPath) > extraResourceFileMaxSize) {
throw new RuntimeException(
s"Extra resource file $fileName exceeds the maximum size limit " +
s"$extraResourceFileMaxSize bytes")
}
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
tempFile.getPath
} catch {
Expand Down

0 comments on commit 287c0dc

Please sign in to comment.