Reprint public account | Lao Liu said NLP
The map of affairs has been a relatively popular direction in the past year. It relies on the feature of reasoning and prediction, and forms a chain of conduction reasoning by building events as the core and causal relationships between events. It was once considered promising.
However, under the current sweep of large models, from the inference side and the construction side, what kind of performance is the big model and whether it has amazing performance.
In order to solve this problem, this article introduces from three perspectives: application search, map construction, and script prediction, and finally introduces an evaluation task based on instruction fine-tuning for your reference.
1. Large models implement common causal chain queries
In the previously released event map demo, there will be functions such as causal query, cause tracing, and probabilistic prediction. Let’s take a look and use the large model to answer directly.
We directly use chatgpt3.5 to query, and the effect is very good. details as follows:
First, the causal query
Enter the cause event:
{
"input_text": "Please list ten answers to "What effects will the earthquake cause" and return it in a list"
}
The result can be obtained:
{
"response": "
1. Earthquakes may cause collapse of buildings, damage to roads, and casualties. \
2. Earthquakes may cause dams to burst, reservoirs to leak, and cause energy and food shortages. \
3. Earthquakes may cause landslides, block roads, and cause traffic paralysis. \
4. Earthquakes can cause tsunamis, causing coastline damage and loss of life. \
5. Earthquakes may cause volcanic eruptions, causing death and property damage. \
6. Earthquakes may cause environmental pollution in local areas and lead to ecosystem collapse. \
7. Earthquakes can lead to government failure, economic collapse, and social unrest. \
8. Earthquakes may cause death and extinction of plants and animals. \
9. Earthquakes can cause massive communication disruptions. \
10. Earthquakes may lead to global warming, causing sea level rise and causing more disasters. "
}
Secondly, causal conduction query, asking questions in the form of a chain, can also be transmitted to the chain.
prompt: Please list ten answers about "What chain reactions will earthquakes produce? Please output them in the form of conduction chains, and the nodes are connected by -> symbols", such as a->b->c->d , returned as a list response: The following are some chain reactions that earthquakes will produce, output in the form of a conduction chain: 1. Earthquake->Building Damage->Casualties->Emergency Rescue 2. Earthquake->fire->explosion->secondary disaster 3. Earthquake -> ground subsidence -> pipeline rupture -> water and gas cut off 4. Earthquake->power interruption->communication interruption->difficulty in information transmission 5. Earthquake->traffic paralysis->difficulty in transporting relief supplies->lifeline blocked 6. Earthquake -> Tsunami -> Coastal flooding -> Damage to crops 7. Earthquake -> groundwater level change -> surface subsidence -> river water level rise 8. Earthquake->river water level rise->dam break->flood disaster 9. Earthquake->Geological structure changes->Groundwater flow direction changes->Water source is polluted 10. Earthquake->social psychological panic->social order out of control->social stability affected
Multi-layer causal reasoning, continue to query, the effect is as follows:
prompt: Please list ten answers about "What chain reactions will earthquakes produce? Please output them in the form of conduction chains, and the nodes are connected by -> symbols", such as a->b->c->d , the length of the link can be as long as possible, but not more than 10 nodes, returned as a list response: 1. Earthquake -> Ground shaking -> Damaged buildings -> Power system outage -> Communications down -> Rescue operations hampered 2. Earthquake -> Ground shaking -> Landslide -> Damage to roads and bridges -> Traffic interruption -> Rescue operations hindered 3. Earthquake -> ground shaking -> tsunami -> damage to coastal areas -> impact on fishery and tourism -> economic loss 4. Earthquake -> ground shaking -> drop in groundwater table -> depletion of groundwater resources -> damage to agricultural production -> reduction in food supply 5. Earthquake -> ground shaking -> fault formation -> formation of new seismic hazard area -> increased risk of earthquake follow-up disasters 6. Earthquake -> ground shaking -> volcanic eruption -> air pollution -> environmental degradation -> public health threat 7. Earthquake -> ground shaking -> interruption of energy supply -> slowdown of industrial production -> reduction of employment opportunities -> decline in social stability 8. Earthquake -> ground shaking -> falling house price -> real estate industry in trouble -> decrease in land transfer income -> decrease in local fiscal revenue 9. Earthquake -> Ground shaking -> Loss of cultural heritage -> Difficulty in protecting cultural heritage -> Decline in the value of historical and cultural heritage 10. Earthquake -> ground shaking -> impaired mental health -> increased mental illness -> increased demand for psychological counseling -> increased pressure on psychological services
Trace the source of the cause, check backwards, and find the cause through the event, the effect is as follows:
prompt: Please list ten answers about "the conduction link of the cause of hair loss, please output it in the form of conduction chain, and the nodes are connected by -> symbol", such as a->b->c->d , the length of the link can be as long as possible, but not more than 10 nodes, returned as a list response: 1. Genes of preference -> Excessive use of heat appliances and hair dyes -> Scalp hair follicle damage -> Hair loss 2. Excessive UV exposure->scalp sunburn->hair follicle damage->hair loss 3. Some drug side effects -> hair follicle damage -> hair loss 4. Hormone imbalance -> hair follicle atrophy -> hair loss 5. Malnutrition -> Insufficient scalp sebum secretion -> Malnutrition hair follicles -> Hair loss 6. Scalp allergies -> itchy scalp scratches -> hair follicle infection -> hair loss 7. Insufficient rest -> Decreased body function -> Hair follicle aging -> Hair loss 8. High mental stress -> affected scalp microcirculation -> malnourished hair follicles -> hair loss 9. Abnormal thyroid function -> slow hair follicle metabolism -> hair loss 10. Skin fungal infection->scalp itching and scratches->inflammatory reaction->hair follicle atrophy->hair loss
Of course, there are also probabilities in the map of affairs, which can also be checked with probabilities:
prompt: Please list ten items about "the conduction link of the cause of hair loss, please output it in the form of a conduction chain, and connect it with the symbol of "node-probability->node", the probability indicates the probability value of conduction, Generally greater than 0 and less than 1. Such as a-0.8->b-0.4->c-0.55->d, the length of the link can be as long as possible, but not more than 10 nodes, and it will be returned as a list response: 1. Stress-0.7->Insomnia-0.6->hormonal abnormality-0.8->hair loss 2. Age-0.6->male hormone level rises-0.75->hair follicle becomes sensitive-0.5->hair loss 3. Unbalanced diet-0.4->vitamin deficiency-0.7->hair follicles cannot grow normally-0.6->hair loss 4. Hair dye or perm -0.5->hair quality deteriorates-0.7->hair follicles gradually lose growth ability-0.8->hair loss 5. Facial skin lesions-0.6->infection of scalp skin and hair follicles-0.7->damage to hair follicles-0.65->hair loss 6. Drug abuse-0.45->hair loss side effects-0.8->hair follicle damage-0.7->hair loss 7. Lack of sleep - 0.55 -> poor blood circulation in the scalp - 0.75 -> lack of nutrition in the hair follicles - 0.5 -> hair loss 8. Unbalanced scalp sebum secretion-0.6->clogged pores-0.4->insufficient oxygen supply to hair follicles-0.7->hair loss 9. Excessive scalp friction-0.35->damaged hair follicles-0.8->short hair growth-0.6->hair loss
2. Causal extraction, structured and scripted prediction based on large models
Let’s now take a look at the two important links in the construction of the event graph, causal extraction and event structuring, and use script prediction on the application side for experiments. Note: The extraction task uses chatglm6b for experiments, and it can be seen that fewshot extraction can be achieved to a certain extent.
1. Causality extraction
1) Prompt design
prompt = "This is a causal relationship extraction task, mainly for the extraction of causal events in financial texts, oriented to the sentence level, the extraction form is (cuase, influence, effect) triplet, and the relationship is only influence Both cause and effect are events in the financial field; for example: China is a big consumer of pork, and the rise in pork prices has attracted great attention from the government. The extraction result is: [(rising pork prices, influence, the government’s high attention)]. Also There is a situation where there are multiple causal events in a text, for example: India’s ban on the import of refined palm oil has increased the demand for crude palm phase oil, and Indonesia’s plan to promote B30 will also increase the demand for crude palm phase oil, which will drive the demand for palm phase oil. Oil prices have been rising in the last few months. The results of the extraction are: [(India's ban on the import of refined palm oil,influence, the demand for crude palm phase oil has increased), (the demand for crude palm phase oil has increased, Influence, the price of palm oil has been rising in the last few months), (Indonesia's plan to promote B30, influence, increase the demand for crude palm oil), (increase the demand for crude palm oil, influence, the price of palm oil is at rising in the last few months)]. Then, given the sentence "{input_text}", the extraction result is: "
2) Extraction effect
Given the sentence: “Cultural paper: students return to school and the demand for party building is improving, and the industry’s prosperity has bottomed out and rebounded. In Q2, affected by factors such as sluggish demand and high inventory, the price of coated paper/double-adhesive paper has increased by 10%/15% month-on-month To 5647/5455 yuan/tun, small and medium-sized businesses in the industry are not profitable,
The extraction result is:
(The prosperity of the culture paper industry has bottomed out and rebounded, influence, students returning to school and the demand for party building is improving),
(Students return to school and the demand for party building is improving, influence, industry prosperity has bottomed out and rebounded),
(Industry boom bottomed out, influence, students resuming school and demand for Party building improved),
(Students return to school and the demand for party building is improving. Influence, industry prosperity has bottomed out and rebounded. Effect, the price of coated paper/double-adhesive paper has increased by 10%/15% month-on-month to 5647/5455 yuan/tun),
(The price of coated paper/double-adhesive paper has increased by 10%/15% month-on-month, influence, small and medium-sized enterprises in the industry are not profitable). "
Given the sentence: “The significant decline in the profitability of the trucking industry is the main reason for the low demand for trucks”
Extraction result:
(The profitability of the freight industry has shrunk significantly, cause, the demand for trucks is not strong)
Given sentence: Urea: As the weather gets warmer, the demand for spring plowing, wheat turning green in the north, and rice fertilizer in the south increases, and the price rises steadily
Extraction result:
(urea price, influence, warmer weather, spring plowing, wheat turning green in the north, increasing demand for rice fertilizer in the south), (The weather is getting warmer, influence, spring plowing, the wheat in the north is turning green, and the demand for fertilizer for rice in the south is increasing), (spring plowing, cause, the wheat in the north turns green, and the demand for fertilizer for rice in the south increases), (Northern wheat reverts to green fertilizer, influence, southern rice fertilizer demand increases, urea price continues to rise), (The demand for rice fertilizer in the south increases, influence, urea prices continue to rise), (The price of urea continues to rise, influence, the weather is getting warmer, spring plowing, the wheat in the north turns green, and the demand for rice fertilizer in the south increases).
2. Event subject-verb-object extraction based on fine-tuning model
1) Prompt design
prompt = This is an event element extraction task, which aims to extract subject-verb-object triples (subject, predicate, object) from the text, where the predicate cannot be empty, and one of the subject or object can be missing. For example, given the text: "The Social Sports Guidance Center of the State General Administration of Sports unites many well-known athletes", the output result is: [(Social Sports Guidance Center of the State Sports General Administration, joint, many well-known athletes)]\
Given the text: " The iron ore stock price plummeted", the output result is: [(iron ore stock price, plummeted,)], now given the text: "{text}", the output result is: "
2) Extraction effect
Given event: “The profitability of the freight industry has shrunk significantly”
Extraction result:
(Freight industry, profitability has shrunk significantly)
Given event: “The profitability of the freight industry has shrunk significantly”
Extraction result:
(crude palm oil, demand, increase)
3. Script reasoning prediction
Script prediction is a commonly used evaluation method of reasoning graph reasoning, which represents multiple choice narrative cloze (MCNC) in the evaluation room. This method was proposed by Granroth-Wilding M et al. In the MCNC task, there is something missing in the complete event chain. Given multiple candidate follow-up events, the model needs to choose the most logical one from multiple candidates.
Among them, the event chain can be a pre-built event chain, and the prompt can be used to reason directly based on this. The following examples are all answered correctly.
Due to English problems, chatgpt is selected for testing below.
1) Data sample
{
'context': ['prisoner forced woman',
'prisoner raped her',
'charges escaped from hospital',
'prisoner held on charges'],
'answer': 'inmate escaped through window',
'candidates': ['inmate escaped through window',
'love follows james', 'they learned from her',
'she turned it', 'it reduce by half',
'games shown in search',
'officials arrived in york'],
}
2) prompt design
'prompt': "it's a Multiple-Choice Narrative Explanation task for predict what happens next, given an start event sequence consists of several single event in an form of 'subject|predicate|object', you should select the correct ending answer from the ending candidates, please return the correct answer directly.\
the start event sequences {context}\
the ending candidates: candidates\
, please select the correct ending event answer:"
3) Actual effect
Enter:
{
'context': ['prisoner forced woman',
'prisoner raped her',
'charges escaped from hospital',
'prisoner held on charges'],
'answer': 'inmate escaped through window',
'candidates': ['inmate escaped through window',
'love follows james', 'they learned from her',
'she turned it', 'it reduce by half',
'games shown in search',
'officials arrived in york'],
}
Output:
Based on the given start event sequences, the most appropriate ending event would be “inmate escaped through window”.
Enter:
{
'context': ['johnson performed ceremony', 'johnson priest of catholic', 'bride met at university', 'she received from university'],
'answer': 'she graduated from harvard',
'candidates': ['he change allow apartments', 'she graduated from harvard', 'organizers urged into 1996', 'company derived from chants', 'back chairman in row', 'services bar in areas', 'she bore in france'],}
Output:
The correct ending event would be: "she graduated from Harvard".
3. Instruction-driven adaptive knowledge graph construction evaluation
Instruction-driven adaptive knowledge graph construction (InstructionKGC) is a method to update and optimize the knowledge graph structure in real time based on user instructions.
The core goal of this method is to adjust the representation of the knowledge graph according to user needs, so as to achieve more accurate and efficient information retrieval and reasoning to meet the needs of efficient knowledge graph construction in an open environment.
In the instruction-driven adaptive knowledge graph construction process, the system (such as a large model) first receives user instructions and recognizes their intentions and needs.
Then, according to the instruction content, the system will extract the corresponding entity-relationship triples based on the input text to construct a knowledge graph. If there is a lack of knowledge, the system will complete the knowledge map according to the completion requirements in the user instruction.
The advantage of this approach is that it enables a custom-planned construction of knowledge graphs. Through adaptive adjustment, the knowledge graph can better meet the needs of different users in different scenarios.
In addition, the instruction-driven adaptive knowledge graph construction method can also improve the scalability of the knowledge graph, enabling it to effectively adapt when facing new data and tasks.
CCKS2023 has set up an evaluation task for this task,
Address: https://tianchi.aliyun.com/competition/entrance/532080/information
2. Mission objective
Extract the corresponding types of entities and relationships according to the instructions entered by the user to build a knowledge graph. Among them, the 10% instruction includes knowledge graph completion, that is, the task requires the model to complete missing triples while extracting triples.
1) Knowledge graph construction task
Enter a piece of text input and instruction (including the relationship type you want to extract), and output all the relationship triples contained in the input in the form of “(head entity, relationship, tail entity)”.
instruction=”Known candidate relationship list: [‘parents’,’spouse’], please extract the possible head entity (Subject) and tail entity (Object) from the following input according to the relationship list, and give The corresponding relation triplet. Please answer in the format of (Subject, Relation, Object).”
input=”On June 23, 2008, Andy Lau and Zhu Liqian registered their marriage in Las Vegas, USA. On May 9, 2012, Zhu Liqian gave birth to her daughter Liu Xianghui in the Hong Kong Sanatorium and Hospital. After Liu Xianghui was born, few saw Liu Xianghui’s face People in the circle said that her daughter’s ears are like Andy Lau, and she has delicate features.”
output=”(Andy Lau, spouse, Zhu Liqian), (Liu Xianghui, parents, Andy Lau), (Liu Xianghui, parents, Zhu Liqian)”
2) Knowledge map completion task
In the case of inputting miss_input (a piece of text is missing in the input, and the input is still smooth) and instruction, the model can still complete the missing triplet and output the output. Below is an example:
instruction=”Known candidate relationship list: [‘parents’,’spouse’], please extract the possible head entity (Subject) and tail entity (Object) from the following input according to the relationship list, and give The corresponding relation triplet. Please answer in the format of (Subject, Relation, Object).”
miss_input=”On May 9, 2012, Zhu Liqian gave birth to her daughter Liu Xianghui in the Hong Kong Sanatorium and Hospital. After Liu Xianghui was born, a few insiders who have seen Liu Xianghui’s face said that her daughter’s ears resemble Andy Lau, and she has delicate features.”
output=”(Andy Lau, spouse, Zhu Liqian), (Liu Xianghui, parents, Andy Lau), (Liu Xianghui, parents, Zhu Liqian)”
Although miss_input does not contain the text “On June 23, 2008, Andy Lau and Zhu Liqian registered their marriage in Las Vegas, USA.”, but the model can fill in the missing triples, (Andy Lau, spouse, Zhu Liqian), (Liu Xianghui, parents, Andy Lau).
3. Task data format
In the release stage of the preliminary training set and verification set, about 5,000 pieces of knowledge graph data are released as the training set. These 5,000 pieces of data are weakly supervised data that use Wikidata to perform remote supervision and annotation on Chinese Wikipedia, so there may be some errors.
{
"id": 12125,
"category": "Event",
"instruction": "Known candidate relationship list: ['military conflict', 'participant', 'type'], please extract the possible head entity (Subject) and tail entity (Subject) from the following input according to the relationship list Entity (Object), and give the corresponding relation triplet. Please answer in the format of (Subject, Relation, Object).",
"input":"The Battle of An Loc was a battle in the latter part of the Vietnam War. It was one of the battles of the North Vietnamese Easter Offensive launched by the Vietnamese People's Army and the Viet Cong against South Vietnam from April 13 to July 20, 1972. In the end, the 5th and 9th Divisions of the Viet Cong who invaded south were repelled by the 5th Division of the South Vietnamese Army and defended An Loc.",
"output": "(Vietnam War, Military Conflict, Vietnam People's Army), (Vietnam War, Participants, North Vietnam), (Vietnam War, Participants, Viet Cong), (An Loc Battle, Type, Vietnam War), ( Easter Offensive, Genre, Vietnam War)",
"kg":[
[
"Vietnam War",
"military conflict",
"Vietnamese People's Army"
],
[
"Vietnam War",
"participant",
"North Vietnam"
],
[
"Vietnam War",
"participant",
"Viet Cong"
],
[
"Battle of Anlu",
"type",
"Vietnam War"
],
[
"Easter Offensive",
"type",
"Vietnam War"
]
]
}
Summary
This article’s main event map is a relatively popular direction in the past year. It relies on the feature of reasoning and prediction, and through the construction of events as the core, causality and other relationships between events to form a chain of conduction reasoning, it was once considered promising.
This article introduces it from three perspectives: application search, graph construction, and script prediction, and finally introduces an evaluation task based on instruction fine-tuning.
It can be seen that whether it is reasoning or search, large models have strong substitutions on the map of events, especially in the reasoning stage. The current COT capability is the charm of large models.
However, as Lao Liu often said before, there may be a bigger way out if the map of events is well developed towards the event ontology and event schema.
References
1. https://tianchi.aliyun.com/competition/entrance/532080/information
OpenKG
OpenKG (Chinese Open Knowledge Graph) aims to promote the openness, interconnection and crowdsourcing of knowledge graph data with Chinese as the core, and promote the open source and open source of knowledge graph algorithms, tools and platforms.
Click Read the original text to enter the OpenKG website.