An Intelligent System for Identifying Influential Words in Real-Estate Classifieds

1 Introduction

Regression is an important data-mining task, where the goal is to predict some numeric value associated with an object using the object features (also called attributes). The numeric value to be predicted is usually called the target attribute. For example, we may wish to predict the price of a real-estate property based on the property features such as the number of bedrooms, number of bathrooms, and its location. We may also wish to predict the probability that a patient would suffer from a certain complication based on the patient’s features, such as age, occupation, and the occurrence of the disease in family.

The majority of the previous work that studied the regression task focused on simple data, where all the attributes are numeric [9, 19, 33]. This paper focuses on regression tasks where the data has a mixture of simple attributes (either numeric or nominal) and textual attributes. We are particularly interested in the problem of quantifying how certain words affect, positively or negatively, the regression task. For example, consider the corpus of real-estate classifieds, which we use as our case study. In this corpus, an object represents a real-estate property and has mix of simple features (such as “number_of_bedrooms”) and textual features (such as “title” and “description”). The problem then is to identify which keywords influence the price of the property. As another example, consider a dataset of patient data, where an object in this case represent a patient and again has a mix of simple features (such as “age” and “gender”) and textual features (such as “physician report”). The goal in this case is to identify which keywords in the “physician report” influence the risk of some disease.

In order to identify which keywords affect the regression positively or negatively, we need to be able to isolate the effect of the textual features from the effect of the remaining simple features. We propose in this paper a two-stage regression model. The first stage of our model uses only simple features of an object to make an initial prediction of the target attribute. The second stage uses the textual features of the same object to refine the initial prediction. For example, consider the real-estate classifieds data where the goal is to predict the price of a real-estate unit. The initial stage would learn to predict the price using only the simple features such as the size of the unit and its location. The second stage will learn to refine the initial prediction using features from the classified text. The decomposition of the regression into two (sequential) stages achieves the following benefits.

Different regression model for each stage: This is important because while linear regression (LR) is convenient to use for the second stage to highlight which keywords are (not) important, LR is not necessarily the best regression model for Stage 1 (which uses simple features).

Intuitive semantics: With such decomposition, the semantics of the LR weights for the second stage are clear: explain, using textual features, why the actual value of the target attribute for a given object is lower than (or larger than) the expected value, as predicted by the regression model in Stage 1 (which relied only on simple attributes).

We conducted the analysis on four corpora of real-estate classifieds. ^[1] We show that our proposed approach improves the prediction of a property price, compared to simple features alone. We also illustrate how the two-stage model can be used to identify keywords that affect the price negatively or positively. Table 1 shows an example classified that was automatically highlighted (annotated) using our proposed system. For instance, the words and represent two neighborhoods (in Dubai, United Arab Emirates), with the being the much more prime location. Also, the words are examples of words that affect the price positively as they reflect desirable features.

Table 1:

Sample Classified with Important Words Highlighted.

To summarize, the contributions of this paper are as follows:

• Proposing a two-stage regression model that exploits the textual features to improve the prediction when the data contain a mix of simple and textual attributes.

• Showing how our proposed model can be used to annotate and highlight keywords that affect the prediction positively or negatively (therefore producing the first system for automatically recognizing and highlighting important textual terms in real-estate classifieds).

• Collecting four corpora of real-estate classifieds and using the four datasets to evaluate our proposed model against three different regression models.

The rest of the paper is organized as follows. The next section explains our two-stage regression model and illustrates how it can be used for identifying important keywords. We then describe the domain we use as our case study, real-estate classifieds. This is followed by the evaluation and analysis of our proposed approach using the collected data. We then discuss the related work and conclude our paper.

2 Two-Stage Regression Model with Text Mining

Figure 1 illustrates the process we propose to analyze mixed data. The main idea is to use a two-stage regression model. The first stage (Stage 1 regression model in the figure) attempts to predict the target attribute using only simple features. The second stage attempts to predict the remaining difference between the actual value of the target attribute and the initial prediction from Stage 1 using textual features.

Figure 1:

The Data Mining Process We Used to Build the Two-Stage Regression Model.

The first stage attempts to predict the target attribute using only the simple features. The second stage attempts to predict the difference between the actual value of the target attribute and the initial prediction from Stage 1.

The following subsections describe the different components of our proposed approach in further detail. ^[2]

2.4 Applying the Model

Figure 2 illustrates how the generated two-stage regression model can be used to predict the value of the target attribute. Each stage predicts a component of the value, and the two components are summed to generate the final prediction.

Figure 2:

The Data Mining Process to Apply the Two-Stage Regression Model.

Each stage predicts a component of the value, and the two components are summed to generate the final prediction.

As we also show in Table 1, the weights of the LR model of Stage 2 can be used to highlight important keywords. Each word is assigned three color components – red, green, and blue. By default, all color components is assigned the value of zero (black color). If a word has a non-zero weight in the LR model of Stage 2, then the word color is changed as follows. If the weight is positive, then the weight is assigned to the blue color component. If the weight is negative, then the absolute value of the weight is assigned to the red color component.

3 Case Study: Real-Estate Classifieds

Real-estate classifieds constitute an integral part of the real-estate market. A real-estate classified provides a concise description of a real-estate unit that is available either for rent or sale. Traditionally, such classifieds were publicized using printed materials such as newspapers or dedicated classifieds periodicals. The Internet revolutionized the classifieds business, as with other advertisement sectors. The large reduction in price, coupled with efficient and quick search service, removed the restriction on the size of classifieds (from a maximum of few tens of words to practically unlimited textual description) and allowed the inclusion of multimedia images and videos. This revolution can be verified through the valuation of Zillow, one of the biggest websites that specializes in posting real-estate classifieds, at 50 billion USD. ^[3] Although at some point, it was feared that such a new trend in advertising properties may threaten the profitability of traditional brokerage companies, many brokers have adapted their operation to exploit what the web has to offer, and they integrated the web technology into their process [7].

Nowadays, many large brokerage companies develop their own websites to list their properties, in addition to listing their properties on third-party web portals. In the United Arab Emirates, where the real-estate market constitutes 12.5% of the GDP, ^[4] several major website portals target real-estate classifieds. Examples include Gulf News Ads (www.gnads4u.com), Dubizzle (www.dubbizle.com), Bayut (www.bayut.com), and Property Finder (www.propertyfinder.ae). Furthermore, major real-estate brokering companies list classifieds on their own websites, such as Better Homes (www.bhomes.com) and Hamptons (www.hamptons.ae).

Despite the importance of real-estate classifieds, there has been little work that analyzed such data. In particular, most of the previous work (that applied data-mining techniques to real-estate data) focused on simple attributes such as number of bedrooms, area, location, etc. [12] (a broader view of related work is given in Section 5). For example, Zillow website provides Zestimate service, which attempts to automatically valuate a real-estate unit. The service relies on history of previously sold houses, using only simple features [16].

However, for classifieds, the unstructured and ungrammatical ^[5] textual attributes (the classified title and description) are important components of a classified that should not be ignored as they may encapsulate less common features (such as “remodeled house”, “near X hotel”). While many websites try to capture less common and ad hoc features in a form of a checklist that ad posters can mark, it is hard to account for all possible features in such manner, particularly rare features. Also, some ad hoc features may appear suddenly over time (e.g. due to the opening of a new park or a hotel).

Two key problems are faced by humans posting real-estate classifieds:

• What is the fair price of a real-estate unit that should be advertised in a classified?

• How to write an effective title and description of a classified?

Answering these questions will be valuable for real-estate brokers and home owners who directly post classifieds. Predicting a more accurate price, for a real-estate unit, that takes into account the textual unstructured data (not just the structured data) will prevent the stakeholder from overestimating or underestimating the price. Furthermore, by identifying the important keywords, the stakeholder can refine the unit description and title to better reflect the price being asked.

We extracted our data from major websites that post online residential real-estate classifieds in the United Arab Emirates. The data was collected ^[6] over the period from 17th of September to 6th of October 2015, using our own web crawler. ^[7] The collected data contained both apartments (flats) and villas (houses) that are offered for either sale or rent. A total of 20,600 records were extracted. Table 2 illustrates the extracted features.

The extracted data was then cleaned as follows. Duplicate classifieds (identical title and description) were removed to end up with 19,644 records. Then, outliers were removed as follows. First, the average rent/price per bedroom was calculated by dividing Price over Beds+1 (the added one was needed in case of a studio apartment that has value Beds=0). Records with unreasonably low/high price were removed according to the following conditions. For the rental properties, we only accepted the properties that satisfy the condition 100,000≥Average rent≥10,000. ^[8] The properties that are up for sale needed to satisfy the condition 1,000,000≥Average price ≥100,000. This resulted in a further reduction in the number of records to 16,008 divided as follows:

• Apartments for rent: 8192 records;

• Houses for rent: 2105 records;

• Apartments for sale: 4599 records;

• Houses for sale: 1112 records.

Then, the location attribute, which is nominal, was converted to numerical attributes. This was done by creating a set of binary attributes, one binary attribute for each nominal value of the location. For example, “Dubai Marina” is a possible value of the location attribute. After this step, we have a binary feature “loc_dubai_marina”, which equals 1 if and only if the location attribute equals “Dubai Marina”; otherwise, it equals 0. Finally, all textual features were converted to lowercase.

4 Analysis

To evaluate our approach, we computed two performance metrics: the root mean squared error (RMSE), and the correlation between the predicted price and the actual price. These two measures were computed (for each of the four datasets) using 10-fold cross-validation to compare our two-stage regression model (which incorporated text mining) against the traditional (one-stage) regression model (which relied only on simple features). The comparison was done using three different regression models for the first stage, while LR was used for the second stage. For simplicity, only 1-Gram was used, with T_max=30% and T_min=3%. The list of English stop-words are provided by RapidMiner. ^[9]

Figure 3 summarizes our results for the four datasets and the three regression models of Stage 1. In all cases, the RMSE of the two-stage model is lower than the RMSE of the one-stage model. Furthermore, and except for the case of using LR in the Selling House data set, the confidence interval of the two-stage model does not include the mean RMSE of the one-stage model. In the remainder of this section, we discuss in more detail the results achieved when different regression models are used for Stage 1.

Figure 3:

Summary of Results for the Four Data Sets (Selling House, Selling Apartment, Renting House, Renting Apartment) and the Three Regression Models of Stage 1 (LR, ANN, and SVMR).

Our approach (two-stage model) achieved lower (better) RMSE across all data sets, regardless of the underlying regression model in Stage 1.

Tables 3 and 4 summarize the results when LR was used for the first stage. The RMSE is reduced and the correlation increased. It is worth noting that the RMSE is higher and the correlation is lesser for houses when compared to apartments. This is due to the larger variety of houses even within the same neighborhood, in addition to the smaller number of house classifieds (compared to apartments). Furthermore, the RMSE for selling (an apartment or a house) is significantly higher than the RMSE for renting. The RMSE is sensitive to the scale of the target attribute we are trying to predict. The annual rent is much less than the selling price, hence the significant difference in RMSE, regardless of the dataset size. For example, a one bedroom may have annual rent of 80,000 AED while being sold for 1,000,000 AED or more. Nevertheless, RMSE is suitable for comparing the relative performance of different techniques on the same dataset (rather than across datasets), which is our goal.

Tables 5 and 6 show similar results when we used ANN [13] for Stage 1. The tables show again clear and statistically significant improvements when using our two-stage approach against the traditional one-stage approach (that also uses ANN). The RMSE is reduced, and the correlation increased in all four datasets. It is worth noting that across all datasets, the performance of ANN is worse than LR; however, the use of text mining in Stage 2 still improved performance.

Tables 7 and 8 show similar results when we used SVMR [26] for the first stage of our approach. Similar to the previous regression models we evaluated, the tables show clear improvements when using our two-stage approach against the traditional one-stage approach (that also uses SVMR). Interestingly, using the traditional one-stage approach, SVMR is slightly worse than LR. However, using our two-stage approach and exploiting the textual features of ads, SVMR results in the lowest RMSE in all four datasets with a clear advantage over LR for the Selling House dataset.

To understand why text mining reduced the RMSE, we investigated the LR model in Stage 2 to identify which words affected the price positively or negatively. Table 9 lists a sample of the discovered important words. Few of the words that affected the price were related to the location. While the location attribute in the original dataset ^[10] did specify the location of the unit in a structured manner, only one location was allowed. Textual description, on the other hand, frequently mention nearby prime locations and landmarks. For example, the Palm islands and Downtown are prime locations in Dubai, and therefore affect the price positively in several of the datasets. The Burj refers to Burj Khalifa, the tallest artificial structure in the world, which is a prime landmark with luxurious real estate surrounding it. The Sports city is a new real-estate project that is not fully developed yet (hence the reduction in rent). Most of the words that affected the price negatively made sense, such as deal, offer, road, price, and sale. Interestingly, some of the words affected the price negatively despite representing positive sentiment, such as spacious and nice. This is in contrast to words like amazing, stunning, and beautiful that also represent positive sentiment but affected the price positively. A possible explanation is that words with strong positive sentiment reflect strong confidence in the worthiness of the property and therefore affect the price positively. On the other hand, words with weak positive sentiment reflect weaker confidence in the actual worth of the real-estate property and hence affect the price negatively.

5 Related Work

A variety of machine-learning techniques were proposed to predict numerical (continuous) variables such as cost and time. For example, support vector regression [33] and neural networks [30] were used to predict travel time in transportation. Case-based reasoning was used to estimate the cost of road pavement projects [6]. Neural networks and support vector machines were used to predict electricity prices [32]. All the above approaches, however, focused on numerical attributes and did not handle textual features and only used a single-stage model. While an early regression model did support mixture of numeric and categorical attributes [8], the model assumed fixed uniform weights for all categorical attributes (unlike our proposed approach). Unlike regression, the focus of our work, there has been recent work in the classification task that handled mixed data (of categorical and numerical attributes) [23]. This includes an interesting recent work that proposed support-vector-oriented instance selection to improve text classification [29]. Another recent work used text mining to analyze and classify code structures in Android malware [28].

Our proposed two-stage model can be viewed as an ensemble regression technique. Ensemble learning is a process that builds a set of models derived from the same dataset and then integrates these models in some way to obtain the final prediction [19, 21]. However, unlike our proposed model, previous work in ensemble regression assumed only one type of models in any ensemble. For example, the ensemble contains only decision trees. The restriction of one model type per ensemble was necessary to allow the generation of an unbounded number of models. Our two-stage model, on the other hand, allows the model in Stage 1 to be completely different from the model in Stage 2. Furthermore, the vast majority of ensemble methods obtained different fitted models (of the same type) by either training different samples of the data or focusing on different features [19]. Our two-stage model uses different target outputs for the two regression models. An ensemble method that is worth mentioning is iterative bagging [3]. In iterative bagging, the target output of an iteration is the error of the previous iteration (the difference between the target output and the prediction of the previous iteration). This is similar to the manner we devise the target output of Stage 2. However, iterative bagging used the same features and the same model type for all iterations. In contrast, our model uses different set of features and a different model for each stage.

Due to the importance of valuating a real-estate property, there has been extensive research on automatic valuation [2, 4, 15, 18, 24, 25]. Most of that work used hedonic models, which assumed the price can be predicted from a combination (usually linear) of the property’s structured features [10], such as number of bedrooms and the location. Traditional hedonic models were based on human expertise, where the model parameters were hand-coded by experts, unlike our proposed method here, which uses data mining. There has been growing literature on the use of data-mining techniques to analyze real-estate data [5, 11, 12, 14, 17]; however, most of the previous work focused only on simple features and ignored textual features. We review sample of these works in the remainder of this section.

One of the early works [17] used decision tree and neural network techniques to predict the sale price of a house. The analysis used data with 15 numerical features that represent the characteristics of houses plus a categorical feature that corresponds to the address. The dataset consisted of 1000 records that were collected from the house sales transactions in Miami, United States. Unlike our work, the analysis focused only on properties for sale (did not include rentals), used only simple features (no text mining), and relied on a much smaller dataset (compared to our 15,000+ records). A broader analysis was conducted in Ref. [31], covering 295,787 transactions from four cities in the United States. Again, only numerical features were used (although a more extensive set of almost 200 features) and no textual features were used (also despite attempting to predict the price, no performance criterion was reported). A more recent work [12] proposed ANFIS (Adaptive Neuro Fuzzy Inference System) and tested the system using 360 records of past sales properties in Midwest United States. The dataset had 14 numerical features, and again no textual feature was used.

Another research paper [11] focused on studying the prediction of prices of apartments in a city in Macedonia. Among the three data-mining techniques that were applied to a dataset of 1200 sales transactions, the logistic regression (very similar to LR) was found to be the superior in prediction accuracy over decision tree and neural network techniques. Like the other earlier mentioned papers, there was no use of textual data. Some work attempted to add structure to unstructured and ungrammatical data. However, the work required domain knowledge to build a reference structure (model) that can be used to extract the corresponding features [20]. The work also was not applied to real-estate data. Our proposed approach does not require deep domain knowledge (aside from simple data cleansing, the whole process is automated).

The only work we know of that analyzed real-estate classifieds is very recent [27]. That work used textual features along with simple features in one stage and therefore did not offer the flexibility and the semantic meaning as our two-stage model. The previous work also did not provide a clear method for automatically highlighting important words.

6 Conclusion and Future Work

We proposed in this paper a two-stage regression model that uses text mining to analyze data that has a mix of simple and textual attributes. We showed how our proposed model can be used to highlight keywords that affect the prediction positively or negatively. To verify the approach, we used real-estate classifieds as a case study and collected four datasets. Three different regression models were evaluated against our two-stage regression model. The results confirmed that our model reduced the RMSE of price prediction (compared to one-stage models that use only simple features) across the four data sets and against the three different regression models. Furthermore, we introduced the first system for automatically recognizing and highlighting important textual terms in real-estate classifieds.

There are several interesting future directions that we are pursuing. We are currently working on extending our analysis to other datasets. This includes real-estate data from other continents (Europe and America). This also includes medical diagnosis datasets, where our approach can identify important keywords (in the textual diagnosis) that are correlated with the severity of some disease. We are also considering the integration of our system with a named-entity recognition component [1], particularly for identifying locations in ungrammatical text (to improve accuracy).