Tag: hive

[hive] Window function ROW_NUMBER()

Article directory `ROW_NUMBER() ` Example window function ROW_NUMBER() In Hive SQL, ROW_NUMBER() is a window function used to generate row numbers. It assigns a unique row number toeach row in the query result set. The following is the basic syntax of the ROW_NUMBER() function: ROW_NUMBER() OVER (PARTITION BY column1, column2, … ORDER BY column3, column4, […]

Spark On Hive principle and configuration

Table of Contents 1. Spark On Hive principle (1) Why let Spark On Hive? 2. MySQL installation and configuration (root user) (1) Install MySQL (2) Start MySQL settings and start at boot (3) Modify MySQL password 3. Hive installation and configuration (1) Modify Hadoop’s core-site.xml (2) Create hive-site.xml (3) Modify the configuration file hive-env.sh (4) […]

Kafka To HBase To Hive

Table of Contents 1. Create a table in HBase 2. Write API 2.1 Write to hbase in normal mode (write one by one) 2.2 Writing to hbase in normal mode (buffer writing) 2.3 Design mode writing to hbase (buffer writing) 3. HBase table is mapped to Hive 1. Create a table in HBase hbase(main):003:0> create_namespace […]

hive parses illegal json data with special characters

1. Background Sometimes when the front end or back end buries logs, the data in json will be added with double quotes or special characters. When the log is logged, the assembled format will not be normal json data, so we need to include it. JSON data with special characters is parsed into normal JSON […]

High-level application scenarios of Hive SQL functions

HIVE is a common tool for data warehouse processing. Just like the standard SQL syntax in RDBMS relational databases, Hive SQL also has many built-in system functions to meet users’ data analysis needs in different scenarios to improve the efficiency of developing SQL data analysis. We can use show functions to view the functions supported […]

2.paimon-hive-flink integration

Overview The use of default catalog has been completed in the paimon series, based on the file system paimon now supports two types of File system catalog is the default Hive method: Relevant metadata information is stored in hive (actually mysql) and can be accessed directly through hive Today we mainly explain the use of […]

Code design: C++ a serialized Archive class

Serialization is a common function. Unlike databases and configuration files, serialization is generally stored in binary mode without considering human readability. The basic principle of serialization is very simple, just write and write in the same order. The main thing that needs to be considered is how to implement it in the simplest way possible. […]

Makefile update archive

Makefile update archive Review the past and learn the new Remake Target profile members Implicit rules for archiving member targets Update archive symbol directory Dangers when using archives Suffix rules for archive files Previous article: Exercise 10 Implicit Rules of Makefile, Table of Contents | Home Page Review the past and learn the new Exercise […]

Use FlinkCatalog to write kafka data to hive

package com.atguigu.flink.test_hk; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.SqlDialect; import org.apache.flink.table.api.TableEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; import org.apache.flink.table.catalog.hive.HiveCatalog; public class KafkaToHive3 { public static void main(String[] args) { System.setProperty(“HADOOP_USER_NAME”, “atguigu”); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); env.setParallelism(1); env.enableCheckpointing(6000); // to use hive dialect tableEnv.getConfig().setSqlDialect(SqlDialect.HIVE); //Register Hive Catalog HiveCatalog hiveCatalog = new HiveCatalog(“hive”, “default”, “conf”); tableEnv.registerCatalog(“hive”, hiveCatalog); //Create […]